The Information Bottleneck Method[1]

A fundamental problem in formalizing our intuitive ideas about information is to provide a quantitative notion of “meaningful” or “relevant” information. It is argued in this paper that information theory, in particular lossy source compression, provides a natural quantitative approach to the question of “relevant information.” Specifically, the authors formulate a variational principle for the extraction or efficient representation of relevant information. The standard analysis of lossy source compression is “rate distortion theory,” which provides a tradeoff between the rate, or signal representation size, and the average distortion of the reconstructed signal. Rate distortion theory determines the level of inevitable expected distortion, D, given the desired information rate, R, in terms of the rate distortion function R(D). The main problem with rate distortion theory is in the need to specify the distortion function first, which in turn determines the relevant features of the signal. Those features, however, are often not explicitly known and an arbitrary choice of the distortion function is in fact an arbitrary feature selection. In this paper the authors formalized this intuitive idea using an information theoretic approach which extends elements of rate distortion theory. They also derived self consistent equations and an iterative algorithm for finding representations of the signal that capture its relevant structure, and prove convergence of this algorithm.